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The Education Endowment Foundation (EEF) 


Education 
Endowment 
Foundation 


The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to 
breaking the link between family income and educational achievement, ensuring that children from all 
backgrounds can fulfil their potential and make the most of their talents. 


The EEF aims to raise the attainment of children facing disadvantage by: 


e Identifying promising educational innovations that address the needs of disadvantaged 
children in primary and secondary schools in England; 


e Evaluating these innovations to extend and secure the evidence on what works and can be 
made to work at scale; 


Encouraging schools, government, charities, and others to apply evidence and adopt innovations 
found to be effective. 


The EEF was established in 2011 by the Sutton Trust, as lead charity in partnership with Impetus 
Trust (now part of Impetus-The Private Equity Foundation) and received a founding £125m grant from 
the Department for Education. 


Together, the EEF and Sutton Trust are the government-designated What Works Centre for improving 


education outcomes for school-aged children. 
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Research and Communications Manager 
Education Endowment Foundation 

9th Floor, Millbank Tower 

21-24 Millbank 
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About the evaluators 


The evaluation teams were led by John Jerrim (Institute of Education) and Anna Vignoles. 


John’s research interests include randomized controlled trials, access to higher education, 
intergenerational mobility, cross-national comparisons and educational inequalities. He has worked 
extensively with the Programme for International Student Assessment (PISA), and has recently 
returned from working at the OECD. John won the inaugural ESRC Early Career Outstanding Impact 
award and has just received an ESRC grant to study cross-national comparisons of educational 
attainment and social mobility. 


Anna Vignoles is Professor of Education at the University of Cambridge. She has published widely on 
the impact of school resources on pupil achievement and on the socio-economic gap in pupil 
achievement. Her research interests include issues pertaining to equity in education, school choice, 
school efficiency and finance and the economic value of schooling. Anna is a Research Fellow at the 
Institute for Fiscal Studies and a Visiting Professor at the Institute of Education. Anna has advised 
numerous government departments, including the Department for Education and the Department of 
Business, Innovation and Skills. Anna is also the economist member of the NHS Pay Review Body. 


Professor Richard Cowan (Institute of Education, University of London) provided assessment advice 
on the primary study. 


Contact details: 


Dr. John Jerrim 

Institute of Education, University of London 
20 Bedford Way 

London 

WC1H OAL 


p: 07590 761 755 
e: J.Jerrim@ioe.ac.uk 
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Executive summary 


The programme 


The Mathematics Mastery programme is a whole-school approach to teaching mathematics that aims 
to raise attainment for all pupils and close the attainment gap between pupils from low income 
families and their peers. The programme aims to deepen pupils’ conceptual understanding of key 
mathematical concepts. Compared to traditional curricula, fewer topics are covered in more depth and 
greater emphasis is placed on problem solving and on encouraging mathematical thinking. 


This summary is based on two randomised controlled trials of Mathematics Mastery funded by the 
Education Endowment Foundation. The first evaluation assessed the impact of the programme on 
pupils in Year 1. 83 schools from London and the South East participated in the trial, with a total 
sample of 4,176 pupils. The second assessed the impact of the programme on pupils in Year 7. 44 
schools from London and the South East participated in the trial, with a total sample of 5,938 pupils. 


Both evaluations assessed the programme’s impact in its first year of adoption. In subsequent years it 
was intended that schools would begin to use the approach with older year groups until it was in place 
across all year groups. The education charity Ark provided participating schools with training and 
resources to support the adoption of the programme. 


Separate evaluation reports from the primary and secondary trials are also available on the EEF 
website. 


Key conclusions 


1. This summary is based on findings from two randomised controlled trials conducted in 
English schools between 2011 and 2014. 


2. Onaverage, pupils in schools adopting Mathematics Mastery made a small amount more 
progress than pupils in schools that did not. The effect detected was statistically significant, 
which means that it is likely that that improvement was caused by the programme. 


3. [tis unclear whether the programme had a different impact on pupils eligible for free school 
meals, or on pupils with higher or lower attainment. 


4. Given the low per-pupil cost, Mathematics Mastery may represent a cost-effective change 
for schools to consider. 


5. The evaluations assessed the impact of the programme in its first year of adoption. It would 
be worthwhile to track the medium and long-term impact of the approach. 


What impact did it have? 


On average, pupils in schools adopting Mathematics Mastery made more progress than similar pupils 
in schools that did not adopt the programme. The small positive effect can be estimated as equivalent 
to approximately one month’s additional progress. The effect detected was statistically significant, 
which means that it is likely that that improvement was caused by the programme. 


The programme had a higher impact on pupils in Year 1, who made approximately two additional 
month’s progress on average, than those in Year 7, who made approximately one additional month’s 
progress on average. 
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It is unclear whether the programme had a different impact on pupils’ eligible for free school meals or 
on higher or lower attaining pupils than on higher attaining pupils. 


In follow-up studies, data from SATs and GCSEs should be used to evaluate the medium and long- 
term impact of the programme on different groups of pupils. 


How secure is this finding? 


Overall, the findings related to Mathematics Mastery noted here are judged to be of moderate 
security. They are based on two large randomised controlled trials conducted in English schools over 
the period 2011-2013. Findings from both individual studies were judged to have been of moderate 
security. 


The findings from the individual trials have been combined using an approach called “meta-analysis”. 
Meta-analysis can lead to a more accurate estimate of an intervention’s effect. However, it is also 
important to note that care is needed in interpreting meta-analysed findings. Due to the ages of pupils 
who participated in the individual trials, the headline findings noted here are more likely to be 
predictive of programme’s impact on pupils in primary school than on pupils in secondary school. 


The evaluation team are not aware of any other high-quality evaluations of Mathematics Mastery that 
have been conducted. However, were new evaluations to be published in the future, these should be 
combined with the two studies noted above to provide a more accurate overall impact estimate. 


The findings were substantially lower than the average effects seen in the existing literature on of 
“mastery approaches”. A possible explanation for this is that many previous studies were conducted 
in the United States in the 1970s and 80s, so may overstate the possible impact in English schools 
today. An alternative explanation is that the Mathematics Mastery programme differed from some 
examples of mastery learning previously studied. For example classes following the Mathematics 
Mastery approach did not delay starting new topics until a high level of proficiency had been achieved 
by all students, which was a key feature in a number of many apparently effective programmes. 


How much does it cost? 


The average ‘per pupil’ cost of the intervention is estimated to be around £131 per year for primary 
school pupils and around £50 per year for secondary school pupils, in the first year, with per pupil 
costs likely to reduce in future years in both cases. 


Effect size Estimated 
(95% confidence months’ Evidence strength* 
intervals) progress 


Number of pupils 


ecu (schools) 


, 10,114 pupils +0.073 
Overall impact (127 schools) (0.00, +0.14) +1 month aaa 
Primary vs. 4,176 pupils +0.10 

comparison (83 schools) (-0.01, +0.21) Sa mOMes aaa 


Secondary vs. 5,938 pupils +0.06 aaaa 


comparison (44 schools) (-0.04 to +0.15) +1 month 


*For more information about evidence ratings, see Appendix A below. 
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Appendix A: Meta-analysis of the Mathematics Mastery 
primary school and secondary school randomised 
controlled trials 


Two Mathematics Mastery trials were conducted simultaneously. The ‘primary school’ trial introduced 
Mathematics Mastery to Year 1 pupils (5/6 year olds) and took place over two academic years 
(September 2012 to August 2013 and September 2013 to August 2014). The ‘secondary school’ trial 
was conducted in the September 2013 to August 2014 academic year, with the focus on Year 7 pupils 
(11/12 year olds). A summary of this is presented in Appendix Table A, along with a meta-analysis of 
the results (note that the primary and secondary school trials have been given equal weight). 


The reported effect size is similar across the two trials (0.10 for primary school and 0.06 for secondary 
school) though neither individually reaches statistical significance at the five per cent level. Precision 
is increased, however, when information is combined across the two. Indeed, the pooled effect size of 
0.073 is just significantly different from zero at conventional thresholds. Overall, these results support 
the conclusion that even a one year dose of the Mathematics Mastery intervention leads to a small 
(yet potentially cost-effective) improvement in children’s maths test scores. 


Table A. Meta-analysis results 


asia Yed aTefe)| lea Yes aTeXe)| Meta tila 


Number of schools 
School response rate 92 per cent 88 per cent 


Number of pupils 4,176 5,938 10,114 


Pupil response rate 82 per cent 77 per cent 
Effect size 0.099* 0.055 0.073** 
Standard error 0.054 0.046 0.035 
95% confidence interval -0.009 to 0.207 -0.037 to 0.147 0.004 to 0.142 


Notes: Authors’ calculations. Meta-analysis has been weighted by standard error. Overall test scores (pre-specified primary 
outcome) have been reported for both trials. Huber-White adjustments have been made to all standard errors to account for 
clustering at the school level. * and ** indicate statistical significance of effect sizes at the 10 per cent and five per cent levels 
respectively. 


Note on evidence security 


The overall security rating is based on the security ratings of the individual randomised controlled 
trials. The estimate of moderate security (“three padlocks”) is based on the study with the weakest 
individual study, which received three padlocks. For more information about security ratings in 
individual reports, see Appendix A in the primary report and Appendix C in the secondary report. 


Education Endowment Foundation 5 


You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk 
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